GH-32609: [CI] Add type checking infrastructure and CI workflow for type annotations #48618

rok · 2025-12-22T00:15:41Z

Rationale for this change

This is the first in series of PRs adding type annotations to pyarrow and resolving #32609.

What changes are included in this PR?

This PR establishes infrastructure for type checking:

Adds CI workflow for running mypy, pyright, and ty type checkers on linux, macos and windows
Configures type checkers to validate stub files (excluding source files for now)
Adds PEP 561 py.typed marker to enable type checking
Updates wheel build scripts to include stub files in distributions
Creates initial minimal stub directory structure
Updates developer documentation with type checking workflow

Are these changes tested?

No. This is mostly a CI change.

Are there any user-facing changes?

This does not add any actual annotations (only py.typed marker) so user should not be affected.

GitHub Issue: [Python] Type checking support #32609

AlenkaF

Thank you @rok for working on this! Splitting up the main PR sounds like a great idea to me.

I only have one small question, otherwise the changes look good to me.

ci/scripts/python_test_type_annotations.sh

rok · 2025-12-23T10:26:37Z

Thanks for review @AlenkaF ! I think we probably want @raulcd to look at this too before merging?

AlenkaF · 2025-12-23T10:32:04Z

Yes, let's wait for Raul 👍

raulcd

Thanks for the PR @rok sorry it took me a while to review with holidays, release, etcetera

ci/scripts/python_test_type_annotations.bat

ci/scripts/python_test_type_annotations.sh

docs/source/developers/python/development.rst

ci/scripts/python_wheel_validate_contents.py

ci/scripts/python_wheel_xlinux_build.sh

raulcd · 2026-01-13T19:27:53Z

@github-actions crossbow submit wheel-manylinux-2-28-cp311-cp311-amd64

github-actions · 2026-01-13T19:30:08Z

Revision: 28fba3a

Submitted crossbow builds: ursacomputing/crossbow @ actions-8a459250fe

Task	Status
wheel-manylinux-2-28-cp311-cp311-amd64

rok · 2026-01-14T10:41:02Z

compose.yaml

      ARROW_S3: "OFF"
      ARROW_SUBSTRAIT: "OFF"
      ARROW_WITH_OPENTELEMETRY: "OFF"
+      PYARROW_TEST_ANNOTATIONS: "ON"


@raulcd do you think we actually want to be able to toggle these? I'm thinking we don't really need them

I think is worth it if instead of using a separate ci/scripts/python_test_type_annotations.sh we use ci/scripts/python_test.sh and we use the PYARROW_TEST_ANNOTATIONS variable to manage whether we want annotations to be tested or not as part of python testing.
If we decide we want to have two separate scripts I think this env var should be removed.
I lean towards having a single python test script that can allow us to also test annotations but I can find arguments for both so not an issue.

Ok, agreed with your reasoning. I also introduced PYARROW_TEST_ANNOTATIONS to the windows script.

rok · 2026-01-14T10:41:33Z

Thanks for review @raulcd ! I've responded to your points.

raulcd · 2026-01-14T11:41:13Z

compose.yaml

      ARROW_S3: "OFF"
      ARROW_SUBSTRAIT: "OFF"
      ARROW_WITH_OPENTELEMETRY: "OFF"
+      PYARROW_TEST_ANNOTATIONS: "ON"


I think is worth it if instead of using a separate ci/scripts/python_test_type_annotations.sh we use ci/scripts/python_test.sh and we use the PYARROW_TEST_ANNOTATIONS variable to manage whether we want annotations to be tested or not as part of python testing.
If we decide we want to have two separate scripts I think this env var should be removed.
I lean towards having a single python test script that can allow us to also test annotations but I can find arguments for both so not an issue.

raulcd · 2026-01-14T11:42:42Z

ci/scripts/python_wheel_macos_build.sh

+# We first populate stub docstrings and then build the wheel
+python setup.py build_ext --inplace
+python -m pip install griffe libcst
+python ../dev/update_stub_docstrings.py pyarrow-stubs


this is not necessary, right?

Same as #48618 (comment). It's a noop and we don't have to introduce python logic in stub PRs if we do here.

ci/scripts/python_wheel_validate_contents.py

raulcd · 2026-01-14T11:44:04Z

ci/scripts/python_wheel_windows_build.bat

+@REM We first populate stub docstrings and then build the wheel
+%PYTHON_CMD% setup.py build_ext --inplace
+%PYTHON_CMD% -m pip install griffe libcst


Until we have real annotations this isn't necessary, right?

I added running dev\update_stub_docstrings.py, so I suppose it's a noop now and we can keep it to make sure it doesn't fail before introducing actual stubs? This will keep CI logic out of stub PRs.

change bat lint add a popd and nicer logging for windows ReplaceElipsis -> DocstringInserter simplify remove sphinx

rok · 2026-01-22T13:46:53Z

@kou could you take a quick look at this?

kou · 2026-01-23T03:03:13Z

.github/workflows/python.yml

+      - name: Test annotations
+        shell: bash
+        env:
+          PYARROW_TEST_ANNOTATIONS: "ON"
+        run: ci/scripts/python_test_type_annotations.sh $(pwd)/python


I don't object this approach but it may be better that we use pre-commit for type checking. (I think that type checking is one of lint checks.)

That's a nice idea! So instead of a new workflow step you are proposing we move these checks into pre-commit script, correct? I'll change to pre-commit approach.

One thing I wonder about - are there users that run all these checks locally and would not like the new extra jobs?

Oh wait, we need compiled pyarrow to run type annotation checks against and that is not practical for pre-commit.

It's too bad. I reverted back to workflow approach, but only put the annotation job into a single workflow now.

dangotbanned · 2026-01-23T12:25:23Z

python/pyarrow-stubs/pyarrow/__init__.pyi

+"""Type stubs for PyArrow.
+
+This is a placeholder stub file.
+Complete type annotations will be added in subsequent PRs.
+"""
+
+from typing import Any
+
+def __getattr__(name: str) -> Any: ...


IIUC this isn't intended to be permanent.
But I thought you might want to consider adding a note/opening an issue to remove it before release.

Here's some examples of how it can mask runtime errors:

fix(typing): Avoid false positives on nw.__getattr__ narwhals-dev/narwhals#3333

Polars module non existent attributes are inferred as Any pola-rs/polars#24334

Oh, interesting I didn't consider this one! Yes this is meant as a temporary and we would want it out before the release. I'll make a note and issue, thanks for pointing it out!

rok · 2026-01-23T22:25:16Z

Docs failure is from the Pandas issue.

rok requested review from AlenkaF, assignUser, jonkeane, kou and raulcd as code owners December 22, 2025 00:15

github-actions bot added Component: Python Component: Documentation awaiting committer review Awaiting committer review labels Dec 22, 2025

rok force-pushed the pyarrow-stubs-pr1-infrastructure branch from afe4699 to 30017ff Compare December 22, 2025 00:20

This was referenced Dec 22, 2025

GH-32609: [Python] Add type annotations to PyArrow #47609

Open

GH-32609: [Python] Add internal type system stubs (_types, error, _stubs_typing) #48622

Draft

AlenkaF approved these changes Dec 23, 2025

View reviewed changes

ci/scripts/python_test_type_annotations.sh Outdated Show resolved Hide resolved

github-actions bot added awaiting changes Awaiting changes and removed awaiting committer review Awaiting committer review labels Dec 23, 2025

github-actions bot added awaiting change review Awaiting change review awaiting changes Awaiting changes and removed awaiting changes Awaiting changes awaiting change review Awaiting change review labels Dec 23, 2025

raulcd reviewed Jan 13, 2026

View reviewed changes

github-actions bot added awaiting change review Awaiting change review awaiting changes Awaiting changes and removed awaiting changes Awaiting changes awaiting change review Awaiting change review labels Jan 14, 2026

rok commented Jan 14, 2026

View reviewed changes

raulcd reviewed Jan 14, 2026

View reviewed changes

rok added 6 commits January 21, 2026 15:10

further work on dev/update_stub_docstrings.py

74cf0fc

lint

28ebbc2

add click for docstring population

4109bb3

Remove dependencies on click and griffe

08ecb71

fix import paths

a2b522d

add PYARROW_TEST_ANNOTATIONS to AMD64 Windows 2022 Python 3.13

cebbfcd

change bat lint add a popd and nicer logging for windows ReplaceElipsis -> DocstringInserter simplify remove sphinx

rok force-pushed the pyarrow-stubs-pr1-infrastructure branch from 5ce9011 to cebbfcd Compare January 21, 2026 14:10

kou reviewed Jan 23, 2026

View reviewed changes

github-actions bot added awaiting changes Awaiting changes and removed awaiting change review Awaiting change review labels Jan 23, 2026

dangotbanned reviewed Jan 23, 2026

View reviewed changes

move check to pre-commit

b5f357b

github-actions bot added awaiting change review Awaiting change review and removed awaiting changes Awaiting changes labels Jan 23, 2026

change pre-commit, add note

577068b

rok mentioned this pull request Jan 23, 2026

[CI] Remove __getattr__(name) -> Any from __init__.pyi before next release #48970

Open

github-actions bot added awaiting changes Awaiting changes and removed awaiting change review Awaiting change review labels Jan 23, 2026

change pre-commit script

27a4a65

github-actions bot added awaiting change review Awaiting change review awaiting changes Awaiting changes and removed awaiting changes Awaiting changes awaiting change review Awaiting change review labels Jan 23, 2026

Revert from pre-commit to ci/scripts/python_test_type_annotations.sh

6f2f175

github-actions bot added awaiting change review Awaiting change review awaiting changes Awaiting changes and removed awaiting changes Awaiting changes awaiting change review Awaiting change review labels Jan 23, 2026

GH-32609: [CI] Add type checking infrastructure and CI workflow for type annotations #48618

Are you sure you want to change the base?

GH-32609: [CI] Add type checking infrastructure and CI workflow for type annotations #48618

Conversation

rok commented Dec 22, 2025 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

AlenkaF left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

rok commented Dec 23, 2025

Uh oh!

AlenkaF commented Dec 23, 2025

Uh oh!

raulcd left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

raulcd commented Jan 13, 2026

Uh oh!

github-actions bot commented Jan 13, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rok commented Jan 14, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rok commented Jan 22, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rok commented Jan 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

rok commented Dec 22, 2025 •

edited by github-actions bot

Loading

rok commented Jan 23, 2026 •

edited

Loading